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FOREWORD 


This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  reading  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  in  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 


vii 


Preceding  page  blank 


T&E  GUIDELINES  FOR  COMMON  TEST  GEAR 


The  checklist  items  presented  here  are  specifically  applicable  to 
common  test  gear  testing  and  evaluation.  It  is  suggested  that  the  user  of 
this  volume  also  refer  to  the  Report  of  the  Defense  Science  Board  on  Test 
and  Evaluation  which  contains  general  checklist  items  also  applicable  to 
this  system  T&E  program.  The  checklist  items  presented  here  are  organized 
into  time  phases  of  the  acquisition  process  oriented  to  the  DSARC  cycle. 

The  checklists  cover  various  aspects  of  the  major  activities  that 
should  be  underway  during  a  given  time  period.  Hence,  a  checklist  might 
cover  the  (1)  evaluation  of  work  that  occurred  in  the  previous  phase, 

(2)  conduct  of  tests  planned  in  the  previous  phase  and  executed  in  the 
subject  phase,  and  (3)  plans  and  other  preparatory  actions  for  test  sche¬ 
dules  to  be  conducted  in  a  subsequent  phase.  For  reasons  such  as  this, 
items  on  some  subjects,  such  as  development  test  plans,  may  appear  in  more 
than  one  phase.  In  addition,  since  the  Services  and  the  DSARC  have  flexi¬ 
bility  in  deciding  how  rapidly  to  progress  in  the  validation  phase,  there 
may  be  cases  where  the  Request  for  Proposals  (RFPs),  proposal  evaluations, 
source  selections,  or  contract  negotiations  may  occur  after  the  DSARC 
approves  full-scale  development  instead  of  before.  For  this  reason,  it 
is  recommended  that  previous  checklists  in  the  Validation  Phase  be  re¬ 
viewed  when  entering  the  Full-Scale  Engineering  Development  Phase.  The 
following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceptual  Phase  of  the  acquisition  of  the  system. 
This  phase  (often  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  de.velopment  of  a  weapon  system  con¬ 
cept  that  offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  for  in  DoD  Directive  5000.1  specifically,  the 
objectives  of  this  phase  should  be: 
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1.  To  verify  that  there  is  a  military  need  for  the  proposed 
system. 

2.  To  demonstrate  that  there  is  a  sound  physical  basis  for  a  new 
weapon  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need. 

4.  To  show  that  the  proposed  solution  is  superior  to  its  com¬ 
petitors  in  terms  of  potential  effectiveness,  probability  of 
success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  need  to 
show  that  it  is  better  to  start  advanced  development  now 
rather  than  to  wait  for  future  technological  improvements. 

6.  To  identify  the  key  risk  areas  and  critical  issues  that  need 
to  be  resolved  before  full-scale  development  is  initiated. 

The  most  important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  time  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSARC  recommends  full- 
scale  development  of  the  system). 

While  these  objectives  are  uot  spelled  out  in  the  DoD  Directive  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm: 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of 
development  costs,  preliminary  estimates  of  life  cycle  costs 
and  potential  benefits  in  context  with  overall  DoD  strategy  and 
fiscal  guidance. 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  and  solutions  are 
in  hand. 

4.  Realism  of  the  plan  for  full-scale  development. 
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In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&E  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the 
T&E  interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must 
Include  adequate  guidance  to  ensure  that  sufficient  resources  and  time  are 
available  so  that  the  engineering  effort  can  properly  support  the  initial 
DT&E  with  hardware,  software,  technical  data,  and  training. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  A  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  incorporated  in  the  engineering  development  contract  as  well  as  the 
DT&E  associated  with  addressing  the  critical  issues  and  areas  of  major 
risk  identified  in  the  DCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in 
conducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  pro¬ 
duction  decision.  By  this  time,  the  system  is  well-defined  and  is 
becoming  a  unique  item  and,  hence,  sound  judgment  must  be  applied  in  using 
these  checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have: 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development 
process  is  complete  and  that  the  design  risks  have  been  mini¬ 
mized  (the  system  is  ready  for  production). 

2.  Demonstrate  that  the  system  will  meet  specifications. 


3 


The  primary  objectives  of  the  IOT&E  should  be  to; 

3.  Assess  operational  suitability  and  effectiveness. 

4.  Validate  organizational  and  employment  concepts. 

5.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  production/deployment 
will  be  recommended  to  the  Secretary  of  Defense. 

The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The 
IOT&E  plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed 
to  evaluate  the  inputs  associated  with  the  primary  objectives.  These 
tests  should  not  be  conducted  until  the  primary  objectives  of  the  DT&E 
have  been  met.  Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E 
major  milestone — the  system  is  ready  for  production — has  been  achieved. 
Close  monitoring  of  the  T&E  Service  activity  is  required  during  the  latter 
stages  of  this  phase. 

SUBSTANTIAL  PRODUCT  ION /DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the  early 
production  items. 

To  enter  the  Production/Deployment  Phase,  the  DSARC  will  have  re¬ 
viewed  the  program  to  confirm; 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of 
production  and  logistic  problems  is  complete. 

•  All  technical  uncertainties  have  been  resolved  and  operational 
suitability  has  been  determined  by  T&E. 

•  The  realism  of  the  plan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 
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2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the 
previous  phase;  however,  much  of  the  testing  is  verification  that  the 
production  system  performance  is  as  expected.  Hence,  most  of  the  items 
in  the  previous  phase  are  appropriate  to  this  phase,  especially  those 
related  to  OT&E. 
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CONCEPTUAL  AND  VALIDATION  PHASES 


The  prime  objective  of  the  Conceptual  Phase  of  a  Common  Test  Gear  pro¬ 
gram  is  to  determine  that  it  is  needed,  to  justify  it  sufficiently,  and  to 
establish  the  program.  The  genesis  of  the  program  is  usually  built  on  a 
growing  need  for  test  gear  to  replace  and  consolidate  existing  equipments 
in  order  to  overcome  serious  deficiencies,  or  high  costs,  or  to  produce  a 
more  efficient  new  system.  The  new  system  is  usually  based  on  studies, 
IR&D,  exploratory,  or  advanced  development  activities.  In  the  Validation 
Phase,  the  issues  raised  by  the  DCP  or  equivalent  documentation  should  be 
resolved  by  conducting  tests  of  the  systems.  The  development  and  opera¬ 
tional  test  plans  will  be  defined  in  considerable  detail. 

The  design  of  a  test  plan  for  Common  Test  Gear  should  include  the 
following  areas  of  testing: 

(a)  Re liability /Availability  -  An  important  objective  of  these  tests  is 
to  ensure  that  no  software  areas  result  in  reinitialization.  This 
is  usually  neglected  and  yet  it  is  important  to  reduce  the  impact 
of  failures. 

(b)  Serviceability /Maintainability  -  Tests  should  demonstrate  how  prob¬ 
lems  are  determined,  diagnosed  and  repaired  and  the  times  associat¬ 
ed  with  each.  In  the  case  of  software  the  procedures  for  diagnosis 
and  elimination  of  "bugs"  must  be  demonstrated. 

(c)  Compatibility  -  Tests  to  show  the  ability  of  the  user  to 
transfer  from  one  item  to  another  and  continue  to  execute 
the  jobs  he  has  been  executing. 

(d)  Usability  -  Tests  for  evaluating  the  human  factor  characteristics 
which  are  of  special  importance  for  these  applications. 

(e)  Throughput  -  The  plan  should  measure  total  time  required  to  carry 
on  a  standard  series  of  testjs. 

(f)  Capability  -  Tests  to  indicate  the  ability  of  the  product  to  func¬ 
tion  under  various  limits  of  physical  stress  and  environment  as 
well  as  work  load. 

(g)  Security /Integrity  -  Tests  to  show  that  the  data  are  protected. 

(h)  Publications  -  Tests  to  demonstrate  the  manuals  are  understandable, 
workable  and  presented  in  logical  sequence. 
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The  test  and  evaluation  checklist  for  Lhese  phases  contains 
reminders  regarding: 

1.  T&E  Criteria 

2.  T&E  Role  In  Verifying  Effectiveness  Of  Design  Specifications 

3.  Boundary  Sizings 

4.  Critical  Issues 

5.  T&E  Actions  Prior  To  DSARC  I 

6.  T&E  Report  Format 

7.  Reliability  Plan 

8.  Establishment  Of  Test  Criteria 

9.  Software  Testing 

10.  New  Technology  Caution 
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1.  T&E  CRITERIA 

T&E  Criteria  for  selection  from  competitive  designs  should  be 
specified  in  advance  with  critical  issues  identified  for  each 
design. 
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Whenever  competitive  designs  are  under  consideration,  T&E  criteria 
for  selection  should  be  specified  in  advance,  with  critical  issues 
identified  for  each  design.  In  addition  the  same  criteria  should  be 
used  to  make  comparisons  with: 

•  Existing  equipments 

•  Militarized  versions  of  commercial  devices 

•  Ad  hoc  combinations  of  separate  units. 

Evaluation  criteria  should  be  based  on  performance  factors  which  are 
measurable  through  testing.  (Examples  are  reliability,  utilization  rate, 
ease  of  handling,  accuracy  rates.)  A  data  collection  and  evaluation  plan 
should  be  developed  which  describes  the  procedure  to  be  used  in  analyzing 
and  evaluating  the  data  collected. 
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2.  T&E  ROLE  IN  VERIFYING  EFFECTIVENESS  OF  DESIGN  SPECIFICATIONS 

T&E  personnel  should  participate  in  systems  definition  to  protect 
the  intended  system  from  tendencies  to  over  extend  its  claimed 
capabilities. 

Test  and  evaluation  participation  at  systems  definition  should 
verify  the  effectiveness  of  the  spc.cif ication  to  avoid  unjustified  or 
unnecessary  overdesign.  Sophisticated  software  test  gear  that  is  not 
designed  for  anything  particularly  but  for  everything  in  general  is 
almost  always  overspecified. 


} 
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In  some  recent  programs  the  desire  to  "sell"  the  equipment  generated 
exaggerated  claims  of  general  application.  When  the  time  came  to  write 
the  software  to  back  up  the  claims,  the  size  of  the  job  was  so  over¬ 
whelming  as  to  compel  a  complete  redefinition  of  the  range  of  appli¬ 
cability  of  the  common  gear. 
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3.  BOUNDARY  SIZINGS 

Before  DSARC  I,  an  analysis  of  at  least  boundary  sizings  should  be 
made  to  determine  the  most  cost-effective  approach  to  test  equipment. 

It  is  recommended  Chat  no  T&E  plan  be  accepted  that  does  not  include 
analyses  of  at  least  boundary  sizings  to  determine  whether,  for  example, 
"built-in  tests  plus  adequate  spares  and  no  field  maintenance"  is  more 
economical  and  effective  in  the  long  term  than  standardized  and  custom 
test  equipment  in  the  field.  The  concept  of  a  standarized  test  system 
seems  often  desirable;  however,  meaningfulness  of  the  concept  is  not 
always  certain  since  there  are  related  factors  such  as  improved  reli¬ 
abilities,  micro-miniaturization,  speed  and  payload  of  modern  air  trans¬ 
portation,  etc.  which  in  concert  may  make  the  former  approach  to  testing 
more  desirable.  Results  of  the  analyses  should  be  documented  as  appro¬ 
priate  in  the  DCP  or  PM  under  "alternatives." 

Past  experience  is  rot  kind  about  the  success  of  standarized  test 
equipment.  The  time  to  worry  is  before  DSARC  I,  not  after  DSARC  II. 

4.  CRITICAL  ISSUES 

The  major  critical  issues,  especially  those  technological  issues 
relating  to  Common  Test  Gear  should  be  addressed  in  the  T&E  section 
of  the  initial  management  document. 


In  evaluating  the  initial  DCP  or  its  equivalent,  it  is  important  to 
ensure  that  the  T&E  section  (objectives  of  the  tests  to  be  conducted 
during  the  time  period  from  DSARC  I  to  DSARC  II)  address  the  major  critical 
issues,  especially  those  technological  issues  which  have  been  identified. 
For  instance,  in  addition  to  the  concerns  listed  in  the  previous  items, 
issues  to  be  considered  may  be,  among  others: 

(a)  Are  the  results  of  the  tests  proposed  to  be  conducted  on  the 
Common  Test  Gear  directly  related  to  its  final  purpose  and 
not  limited  to  the  determination  of  accuracy,  stability,  etc? 

For  instance,  in  a  device  capable  of  testing  radars  for  MTI 
performance,  will  the  test  equipment  supply  results  that  can 
easily  be  converted  into  cancellation  ratios  and  subclutter 
visibility  limits? 
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(b)  In  Common  Test  Gear,  often  calibrations  are  included  in  the  soft¬ 
ware.  One  should  ensure  that  the  software  is  available,  suffi¬ 
ciently  accurate  for  its  purpose,  and  complete. 

Each  test  should  have  a  single  objective,  if  possible,  and  the  objec¬ 
tive  should  be  simply  stated.  A  plan  for  the  conduct  of  the  test  and  the 
data  collection,  reduction,  and  analyses  must  be  in  sufficient  detail  so 
that  one  can  readily  evaluate  the  performance  of  the  system  and  whether  or 
not  the  test  objective  can  be  met.  A  relationship  between  the  identified 
periormance  parameters  and  the  test  results  should  be  established  prior  to 
the  conduct  of  the  test.  Further,  the  set  of  objectives  for  each  of  the 
tests  should  be  clearly  related  to  the  program  objective  as  defined.  When 
this  relationship  is  not  clear,  amplifying  data  should  be  required. 

5.  T&E  ACTIONS  PRIOR  TO  DSARC  I 

Prior  to  DSARC  I,  a  T&E  plan  for  those  tests  to  be  conducted  prior  to 

DSARC  II  should  he  developed. 

Prior  to  DSARC  I,  sufficient  material  should  be  generated  to  allow  for 
an  initial  evaluation  of  the  overall  T&E  program.  As  part  of  this,  a  test 
and  evaluation  plan  for  those  tests  to  be  conducted  prior  to  DSAHC  II  to 
validate  the  concept  and  hardware  approach  to  the  Common  Test  Gear  should 
also  be  developed.  The  plan  must  include  statements  of: 

(a)  The  critical  issues  and  the  overall  purpose  of  the  test  program. 

(b)  The  class  of  equipments  for  which  the  test  device  will  be  employ¬ 
ed  and  the  mission  and  characteristics  of  these  equipments. 

(c)  The  major  test  objectives  as  related  to  the  missions  of  (a)  above. 

(d)  The  schedule  of  test  milestones — these  milestones  should  specifi¬ 
cally  include  use  of  the  proposed  equipment  on  some  of  the  devices 
for  which  it  is  intended. 

(e)  The  major  resources  required: 

•  Test  environment,  facilities  and  instrumentation 

•  Operational  environment. 

(f)  The  organizations  which  will  conduct  the  test  program. 

(g)  The  analysis  and  evaluation  approach — this  analysis  should  include 
time  required  for  a  test  series,  lists  of  hardware  failures, 
analysis  of  failure  modes. 

(h)  The  degree  to  which  the  service's  independent  test  agency  is 
involved. 
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6.  T&E  REPORT  FORMAT 

A  T&E  reporting  format  should  be  established  and  used  throughout  the 
duration  of  the  program. 


Establish  a  T&E  reporting  format  for  the  program — insist  on  its  use 
throughout  the  duration  of  the  program.  Use  this  to: 

(a)  Establish  a  closed  loop  reporting  and  resolution  process  which 
assures  that  each  test  failure  at  every  level  is  closed  out  by 
appropriate  action,  i.e.,  redesign,  procurement,  retest,  etc. 

(b)  Establish  a  relationship  between  T&E  of  the  Common  Test  Equipment 
and  the  program  offices  of  the  equipments  for  which  the  device  is 
intended. 


7.  RELIABILITY  PLAN 

A  study  of  the  test  sample  size  versus  the  confidence  level  for  relia¬ 
bility  of  Common  Test  Gear  should  be  included  in  any  reliability  testing 
plan. 

In  order  to  provide  a  reasonable  confidence  of  the  reliability  of  the 
Common  Test  Gear,  a  large  test  sample  size  might  be  required.  A  study  of 
the  sample  size  versus  the  confidence  level  should  accompany  any  reliability 
testing  plan. 

Remember  that  in  Common  Test  Gear  the  presence  of  undetected  failures  or 
deteriorations  can  be  fatal.  In  these  equipments  reliability  includes  a 
broader  range  of  failures  than  in  most  other  systems. 

8.  ESTABLISHMENT  OF  TEST  CRITERIA 

By  the  end  of  the  systems  definition  phase  for  Common  Test  Gear,  test 
criteria  must  be  established  so  that  tests  may  clearly  determine 
performance  failure  or  success. 

By  the  end  of  the  systems  definition  phase,  test  and  evaluation  should 
make  certain  that  "test  criteria"  are  established  so  there  is  no  question 
as  to  what  constitutes  a  test  and  what  performance  will  constitute  a  success. 

In  Common  Test  Gear  one  should  define: 

(a)  For  what  range  of  equipments  the  test  gear  will  be  employed. 

(b)  What  measurements  will  be  operationally  significant. 
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(c)  What  results  one  must  obtain  to  meet  the  minimum  needs  of 
the  equipments  listed  under  "(a)". 

(d)  What  steps  will  be  taken  if  one  or  another  class  of  failures 
is  found. 

Never  assume  that  Common  Test  Gear  can  be  tested  "per  se"  without  tying  it 
to  its  ultimate  purpose. 

9.  SOFTWARE  TESTING 

Individual  software  units  should  be  tested  prior  to  full-scale  testing 
of  any  software  code. 

Prior  to  full-scale  testing  of  any  software  code,  testing  of  individual 
software  units  (modules)  should  be  conducted  which  requires  machine  execu¬ 
tion  of  every  (or  most)  logical  path  through  that  module.  This  testing 
should  concern  itself  with  the  proper  execution  of  only  that  particular  test 
and,  hence,  the  test  results  should  not  reflect  the  rest  of  the  program, 
but  just  the  information  about  that  particular  module.  The  main  objective 
of  these  tests  is  to  ensure  that  a  module  of  code  will  operate  correctly 
when  it  is  tested  as  part  of  an  overall  software  assembly.  To  achieve  this 
objective,  serial  dependents  within  and  between  tests  should  be  avoided. 
Special  hardware,  or  set  up  conditions  should  be  kept  to  a  minimum  and, 
when  used,  isolated;  more  important,  tests  should  be  designed  so  that  they 
can  be  run  remotely  whenever  possible. 

10.  NEW  TECHNOLOGY  CAUTION 

When  new  technologies  are  used  in  Common  Test  Gear,  the  performance 
may  be  adversely  affected  in  untried  environments.  T&E  should  be 
planned  accordingly. 

It  is  a  natural  tendency  to  use  new  technologies  in  Common  Test  Gear, 
for  instance  to  generate  microwave  signals  from  solid  state  generators. 

In  general,  beware  of  the  performance  of  these  new  technologies  in  untried 
environments!  For  instance,  in  one  case  electronic  gear  was  installed  in 
a  ship  compartment  where  the  field  of  the  ship  radar  was  generating 
spurious  signals  that  made  the  equipment  inoperable. 
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FULL-SCALE  ENGINEERING  DEVELOPMENT  AND  PRODUCTION/ DEPLOYMENT  PHASES 


In  the  Full-Scale  Engineering  Development,  the  Development  Testing  and 
the  10T&E  are  to  be  conducted.  It  is  assumed  that  the  test  plans  prepared 
during  the  previous  validation  phase  will  be  refined  and  the  testing  will 
be  conducted  in  this  third  phase  aimed  at  demonstrating  that  a  substantial 
product  ion /deployment  decision  is  warranted. 

Many  of  the  items  listed  in  this  phase  may  be  more  appropriate  in  the 
earlier  phase  in  some  programs;  for  example,  contract  checklist  items  may 
be  proper  in  the  earlier  phase  if  the  program  is  defined  and  contracts  are 
negotiated  preparatory  for  the  Full-Scale  Development  DSARC.  In  other 
cases,  the  RFP  for  engineering  development  might  not  be  issued  until  after 
the  DSARC  and,  in  this  case,  contracting  checklist  items  may  be  more  appro¬ 
priate  in  the  later  phase. 

Follow-on  OT&E  is  conducted  with  the  early  production  equipment.  The 
lead  time  after  the  production  decision  and  before  the  first  system  is  pro¬ 
duced  is  probably  on  the  order  of  2  years  or  more;  consequently,  the  detailed 
planning  for  follow-on  OT&E  can  probably  wait  until  after  the  production 
decision.  However,  prior  to  that  time  the  basic  plans  should  have  been 
made  for  manning  assignments,  personnel  training,  hardware  and  software 
requirements,  and  facilities. 

The  checklist  includes: 

1.  Government  Furnished  Equipment  and  Facilities 

2.  Interface  Evaluation 

3.  Time  and  Funding 

4.  Human  Factors 

5.  Test  Program  Problem  Indicators 

6.  Type  Personnel  Mix 

7.  Test  Failure  Procedures 

8.  Specification  Verification 

9.  Software  Incompatibility  Tests 

10.  Configuration  Testing 

15  Preceding  page  blank 


11.  System  Perimeter  Measurements 

12.  Limitations  Testing 

13.  Error  Handling  and  Documentation  Validity 

14.  Testing  to  Original  Design  Specifications  and  to  Subsequent 
Modifications 

15.  Simulated  Software  Tests 

16.  Test  Facilities 

17.  Relationship  of  Common  Test  Gear  System  to  the  Supported 
System 

18.  Technical  Maintenance  Data  Package  Testing 

19.  Pilot  Tests 

20.  First  Article  Testing 


1. 


GOVERNMENT  FURNISHED  EQUIPMENT  AND  FACILITIES 


T&E  should  be  concerned  about  the  availability  of  GFE  equipment  as 
specified  in  the  proposed  contract. 


If  there  are  GFE  and  other  government  commitments  in  the  proposed 
contract,  be  concerned  about  the  following: 

(a)  Can  the  gear  with  requirement  performance  be  available  when 
required  for  the  tests? 

(b)  Will  equipment  for  which  the  test  gear  in  intended  be  available? 

(c)  Avoid  contract  terms  on  fixed  price  contracts  that  vaguely 
commit  the  government.  Don't  include,  "government  support 
as  required"  or  "test  facilities  will  be  made  available 
when  needed." 


2.  INTERFACE  EVALUATION 

The  interface  between  the  Common  Test  Gear  and  the  equipment  it 
services  must  be  evaluated  from  several  viewpoints. 

The  use  of  Common  Test  Gear  imposes  constraints  upon  the  design  of 
equipment  that  the  test  gear  is  designed  to  service. 

In  any  T&E  plan  attention  must  be  given  to  this  factor  to  ensure  that 
testing  of  the  common  gear  includes  testing  of  these  features  of  the  service 
equipment.  The  interface  between  the  Common  Test  Gear  and  the  gear  to  be 
tested  should  be  evaluated  from  the  viewpoint  of  complexity,  durability, 
time  required  to  make-up,  and  commonality  with  other  interface  connections. 

3.  TIME  AND  FUNDING 

Analysis  of  the  time  and  funding  required  to  carry  out  the  T&E  plan 
should  not  presume  that  a  series  of  tests  can  be  completed  without 
test  stoppages. 

As  part  of  the  T&E  plan,  provisions  must  be  made  to  ensure  that  time 
and  funding  are  included  to  carry  on  the  necessary  tests.  In  analyzing 
time  and  funding  required  do  not  assume  that  the  series  of  tests  can  be 
carried  on  successfully  as  planned.  When  testing  Common  Test  Gear,  remem¬ 
ber  that  failures  of  the  serviced  equipment  can  stop  the  tests  just  like 
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failures  of  the  test  gear  itself.  A  significant  percentage  of  the  total 
tests  (runs,  trials,  experiments)  should  be  allowed  for  retesting. 

4.  HUMAN  FACTORS 

T&E  should  authenticate  the  human  factors  concepts  embodied  in  the 
proposed  system  design  as  early  as  possible. 

T&E  should  examine  questions  of  safety,  comfort,  appropriateness  of 
man-machine  interfaces,  as  well  as  the  number  and  skill  levels  of  the  per¬ 
sonnel  required  and  the  training  requirements.  The  numbers  of  personnel 
required  should  be  validated  against  both  operational  and  maintenance 
requirements.  Testing  early  versions  in  the  "human  acceptability  and 
compatibility"  environment  is  extremely  important.  This  will  also  help 
to  validate  the  manning  and  training  requirements. 

In  one  development,  a  system  designed  for  rapid  testing  of  a  variety 
of  avionics  devices  was  found  to  be  much  slower  than  originally  intended 
because  the  position  of  switches,  knobs  and  displays  were  placed  so  as  to 
reduce  the  maximum  throughputs  by  factors  of  three  to  four,  thus  eliminating 
the  advantages  claimed  for  the  device  when  the  program  was  approved. 

5.  TEST  PROGRAM  PROBLEM  INDICATORS 

An  early  detection  scheme  should  be  established  to  determine  when  a 
test  program  may  be  underfunded. 

Establish  an  early  detection  scheme  for  government  and  contractor 
management  to  determine  that  a  test  program  may  be  underfunded.  At  this 
time  there  may  be  a  good  possibility  of  recovery.  Some  of  the  indications 
of  trouble  are: 

(a)  Any  repetitive  failures. 

(b)  Major  redesigns  are  suggested  by  test  failures. 

(c)  The  size  of  the  memory  required  in  the  data  processor 
is  increasing  at  an  alarming  rate. 

(d)  A  revision  of  schedule  or  incremental  funding  that  exceeds 
the  original  plan.  Predicted  downstream  recovery  may  not 
have  a  realistic  basis. 
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(e)  Any  relaxation  of  basic  requirements  such  as  time  to  test, 

test  accuracy,  types  of  equipment  to  be  serviced,  etc. 

0.  TYPE  PERSONNEL  MIX 

A  mix  of  personnel  with  different  type  backgrounds  is  needed  through¬ 
out  the  T&E  program.  The  operator  of  the  equipment  to  be  tested  (by 

the  Common  Test  Cear)  must  be  kept  in  the  T&E  loop. 

Testers,  evaluators,  and  operators  have  quite  different  backgrounds 
and  needs  which  affect  the  T&E  of  the  Common  Test  Gear  system.  Each  has 
a  different  approach  which  has  merit  and  utility  at  almost  all  points  in 
the  T&E  program.  A  mix  of  these  types  is  needed  throughout  the  program. 
Early  in  the  program,  the  lead  emphasis  should  be  from  the  tester,  shifting 
to  the  evaluator  and  finally  the  operator,  but  at  all  times  all  parties  and 
their  needs  should  be  coordinated. 

In  the  case  of  Common  Test  Gear  the  operators  of  the  equipment  to  be 
tested  will  be  final  judges  of  the  operational  suitability  of  Common  Test 
Gear.  Make  sure  that  the  T&E  program  includes  these  operators  in  the  loop. 


7.  TEST  FAILURE  PROCEDURES 

Test  plans  should  include  procedures  whereby  a  record  is  kept  of  all 

faixures,  time  lost  and  repair  times . 

The  T&E  plan  should  include  procedures  to: 

(a)  Keep  track  of  all  failures  (even  operator  caused)  whether 
due  to  software  bugs,  failure  of  components,  random 
causes,  improper  design,  etc. 

(b)  Keep  track  of  time  lost  due  to  failures  of  the  serviced 
equipment . 

(c)  Keep  track  of  the  time  required  to  repair  or  fix  every 
hardware  box  or  software  module. 


8.  SPECIFICATION  VERIFICATION 

Tests  need  to  be  conducted  to  verify  that  the  specified  functions 
associated  with  the  Common  Test  Gear  match  the  programmed  functions. 


19 


Specifically,  these  tests  should: 

•  Verify  that  the  explicit  functional  specification  has  been 
correctly  implemented. 

•  Verify  that  the  explicit  logic  specifications  have  been 
correctly  implemented. 

•  Verify  that  all  programmed  functions  have  been  fully  specified. 

9.  SOFTWARE  INCOMPATIBILITY  TESTS 

Tests  should  be  conducted  which  identify  areas  where  the  product 

has  incompatibilities  with  existing  support  and  standards. 

Since  a  programming  functional  specification  describes  what  the 
product  does,  why  it  is  being  provided,  and  how  it  is  to  be  used,  tests 
should  be  conducted  which  identify  areas  where  the  product  has  incom¬ 
patibilities  with  existing  support  and  standards.  These  should  include 
incompatibilities  in  source,  command,  or  control  language.  The  tests 
should  further  identify  changes  in  usage  or  human  factors,  as  well  as 
allow  for  investigation  of  comparison  techniques  required  to  resolve 
these  incompatibilities.  Measures  of  performance  should  include  execution 
time  and  response  time,  real  and  auxiliary  storage  estimates. 

10.  CONFIGURATION  TESTING 

Configuration  testing  on  Common  Test  Gear  is  required  to  verify  that 

the  product  operates  within  the  hardware  and  software  systems  that 

support  it  relative  to  hardware  configuration. 

Tests  should  exercise  the  hardware  and  the  software  code  and  should 
exercise  the  code  on  various  hardware  configurations  to  verify  that  there 
are  no  hardware  deficiencies  relative  to  software.  Configurations  tests 
are  needed  to  verify  that  the  function  is  viable  in  the  support  software 
environment  such  as  sequential  processing  and  multi-processing. 
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11.  SYSTEM  PARAMETER  MEASUREMENTS 

Testing  should  be  designed  to  show  that  the  combination  of  the 
appropriate  software  package(s)  and  hardware  (both  that  of  the 
test  gear  and  the  system  to  be  tested)  is  such  that  all  para¬ 
meters  of  the  system  under  test  are  measured. 

Tests  should  exercise  the  hardware  and  the  software  code  on  various 
configurations  to  verify  that  there  are  no  hardware  deficiencies  relative 
to  software. 

12.  LIMITATIONS  TESTING 

Tests  should  be  conducted  to  define  and  verify  system  limitations . 

As  soon  as  possible,  limitations  testing  should  be  conducted  on  a 
Common  Test  Gear  system  to  verify  that  the  product  limits  are  correctly 
stated.  To  do  this,  the  product  should  be  tested  outside  the  limit,  at 
the  limit  and  within  the  limit.  Relative  to  external  limits,  items  that 
should  be  investigated  are  capacity  and  quantitative  constraints  stated 
in  the  specifications,  such  as  the  size  of  the  record,  depth  of  test,  etc 
Internal  limits  should  also  be  investigated  including  table  size,  queue 
limits,  etc. 

13.  ERROR  HANDLING  AND  DOCUMENTATION  VALIDITY 

The  error  handling  facility  of  the  Common  Test  Gear  system  should 
be  tested  for  adequacy  and  verification  of  performance.  Documen¬ 
tation  associated  with  the  system  should  be  tested  for  validity 
and  accuracy. 

Tests  need  to  be  conducted  on  Common  Test  Gear  to  verify  that  the 
error  handling  facility  of  the  product  operates  satisfactorily,  and  that 
these  facilities  are  sufficient  for  the  type  of  errors  which  are  likely  to 
occur.  In  order  to  do  this,  one  can  force  the  system  to  produce  every 
known  type  of  error  message  and  verify  the  accuracy  and  clarity  of  each. 

Testing  should  provide  for  investigation  of  errors  from  the  operator, 
the  source  language  and  hardware  failures.  It  is  important  that  the  publi¬ 
cations  associated  with  the  Common  Test  Gear  be  accurate  and  clear.  Tests 
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should  be  conducted  to  verify  the  validity  of  the  publication;  for  example, 
the  figures  and  tables  concerning  functions  appearing  in  the  documentation. 

14.  TESTING  TO  ORIGINAL  DESIGN  SPECIFICATIONS  AND  TO  SUBSEQUENT 
MODIFICATIONS 

T&E  should  not  only  verify  original  design  specifications  but  also 
assess  the  impact  of  any  new  modifications  into  the  system. 

Test  and  evaluation  should  ensure  that  original  specifications  are 
met  including  MTBF,  duration  of  unscheduled  interruptions,  and  service 
hour  criteria.  If  and  when  modifications  are  introducted  into  the  system, 
additional  evaluation  should  take  place  to  measure  the  impact  of  these 
changes  as  they  directly  relate  to  the  original  objectives.  Since  the 
test  gear's  own  availability/maintainability  is  a  key  ingredient  to  the 
overall  economics,  it  is  certainly  appropriate  that  specific  goals  be  set 
for  the  number  of  unscheduled  interruptions  of  the  system  per  month,  the 
duration  of  unscheduled  interruptions,  and  the  maintenance  hours  required. 

15.  SIMULATED  SOFTWARE  TESTS 

Simulated  software  tests  should  be  conducted  prior  to  the  publication 
of  the  programming  logic  specifications. 

Prior  to  the  publication  of  the  programming  logic  specifications,  a 
simulated  test  should  be  conducted  to  assure  that  all  interfaces  have  been 
considered  and  accounted  for,  and  that  the  detail  logic  is  error  free. 

This  simulation  should  include  participation  of  the  designer,  the  coders, 
the  testers,  and  publication  planners.  A  set  of  functional  test  cases 
should  be  prepared  to  determine  how  the  programming  logic  implements  the 
functional  variations  to  be  tested.  The  operating  premise  should  be  that 
the  combination  of  all  functional  test  cases  hits  all  modules.  This  sim¬ 
ulation  should  be  used  to  identify  additional  testing  which  may  be  sensi¬ 
tive  to  the  program's  environment  and  data  structure,  for  example,  buffer 
length,  loop  control,  etc. 
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16.  TEST  FACILITIES 


Prior  to  completion  of  the  development  phase,  test  facilities 
requirements  should  be  ascertained. 

No  later  than  completion  of  the  development  phase,  planners  should 
make  certain  that  there  are  facilities  to  permit  test  sets  to  be  developed 
and  verified  without  inordinate  expense  and  undue  consequences  to  the 
overall  program.  The  need  for  on-line  debugging  and  test  set  validation 
is  unavoidable;  the  complete  absence  of  any  simulation  or  remote  debug 
and  test  capability  is  a  significant  handicap. 

17.  RELATIONSHIP  OF  COMMON  TEST  GEAR  SYSTEM  TO  THE  SUPPORTED  SYSTEM 

T&E  of  Common  Test  Gear  systems  must  be  interfaced  with  the  supported 
prime  weapon  system  at  each  phase  of  the  latter's  development. 

It  is  very  important  that  the  Common  Test  Gear  system  be  tested  and 
evaluated  with  its  prime  weapons  systems  at  each  phase  of  the  development 
of  the  prime  weapon  system.  Test  gear  which  has  been  conceived  as  a  pro¬ 
gram  intended  to  support  a  number  of  major  weapons  systems  must  be  managed 
with  emphasis  placed  on  assessing  how  passing  test  gear  milestones  can  be 
used  to  reduce  the  acquisition  risks  associated  with  the  weapon  systems. 

The  T&E  plan  could  usefully  make  reference  to  the  milestones  of  the  desig¬ 
nated  weapon  systems  to  ensure  that  failure  of  one  program  to  pass  a 
milestone  has  no  serious  effects  on  the  schedule  of  the  other  program. 

18.  TECHNICAL  MAINTENANCE  DATA  PACKAGE  TESTING 

Prior  to  full-scale  production,  testing  must  ensure  that  the  system 
can  be  maintained. 

Prior  to  the  decision  to  go  into  full-scale  production  of  the  system, 
a  complete  technical  maintenance  data  package  including  the  whole  set  of 
software  module  specifications,  must  be  prepared  and  tested  to  ensure  that 
the  system  can  be  maintained.  The  testing  of  this  package  should  be  con¬ 
sidered  first  as  part  of  DT&E  and  then  as  part  of  the  IOT&E  of  the  system. 
Criteria  for  successful  demonstration  of  this  package  should  be  established 
in  both  types  of  tests. 
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19.  PILOT  TESTS 

A  pilot  test  should  be  run  for  the  purposes  of  shaking  down  test 
plans. 

Before  any  operational  tests  for  demonstration  of  operational  suit¬ 
ability  and  effectiveness  is  conducted,  a  pilot  test  should  be  held  with 
the  primary  purpose  of  shaking  down  the  test  plan,  the  instrumentation 
concept  and  the  data  analysis  plan.  A  secondary,  but  vital  purpose  should 
be  to  provide  final  training  for  the  test  participants.  The  pilot  test 
should  be  conducted  sufficiently  prior  to  the  OT&E  so  that  sufficient  time 
is  available  to  make  the  necessary  changes  to  the  OT&E  as  dictated  by  the 
results  of  the  pilot  test. 

20.  FIRST  ARTICLE  TESTING 

Preproduction,  first  article  tests  should  be  planned  and  conducted 
to  confirm  the  adequacy  of  the  equipment  to  meet  specified  perfor¬ 
mance  requirements. 

The  preproduction,  first  article,  testing  and  evaluation  should  be 
designed  and  conducted  to  confirm  the  adequacy  of  such  factors  as  drift 
ol  critical  performance  parameters  with  time,  failure  modes  and  rates, 
etc.  Development  tests,  usually  conducted  first  by  the  contractor  and 
then  by  the  service,  should  be  functional  in  nature  and  at  the  unit  or 
block  box  level,  and  should  be  considered  as  a  service  acceptance  test. 
These  tests  should  be  followed  by  OT&E  which  should  include  known  fail¬ 
ures  of  the  equipment  that  the  test  gear  is  designed  to  support.  For 
example,  test  gear  for  fire  control  systems  should  be  evaluated  as  to  time 
to  detect  and  reliability  of  diagnosing  a  known  failure  in  the  fire  control 
system. 
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